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Background 

-  Why  HRI  is  necessary 

-  Previous  programs,  how  they  feed 
TARDEC’s  HRI  Approach 

-  Program  Methodology 

•  Requirements  analysis/Task  decompos 

•  Ontology/Behaviors  development 

-  Modeling  Environment 

End-to-end  modeling  environment 

•  Constructive  modeling/simula  ion 

•  Component/system/vehicle  modeling 

•  Virtual  and  HWITL  simulation 

-  Technology  Exploration 

•  Multi-model  devices 

•  Interfaces 
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Problem  Definition 
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Mobility 


Payloads 


Missions 
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The  soldier  has  an  ever  increasing  task  loa< 


...and  still  must  perform  his  primary  mission! 


Class  i  UAV 


FC  S  JCV 


While  operating 
Mounted 


Operating  with  varying 


Manned  Systems 


Unattended  1  “ 

Ground  Sensors  ^ 


Land  Warrior 


SUGV 


Interact  with: 

-  Other  soldiers 
Manned  systems 


-  Unmanned  systems 

• - Grou n d  vehicles 


Air  vehicles 
Ground  sensors 


Class  II  i'll  I  UAV 


Problem  Definition  (cont 
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*  Many  different  unmanned  systems  in  existence 

*  Each  system  has  developed  unique  Interface 

-  Integrator  specific  -  unique  solution  to  unique  problem 

-  Typically  engineering  solutions  -  not 

*  Lack  of  standardization  for  WMI’s 

*  Increased  complexity  and  diversity  of  systems  and 
interfaces  requires: 

-  specialized  training 

-  Retraining/familiarization  when  moving  between  systems 


Undertime  critical  life/death  situations,  this  is 

to  the  soldier 


unacceptable 


U.S.  ARMY  TANK-AUTOMOTIVE  RESEARCH  DEVELOPMENT  AND  ENGINEERING  CENTER 


September  2002  ASB  Study 

Findings 
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No  existing  program  is  systematically  addressing  the 
challenges  of  humans  and  complex  unmanned  systems 
interactions. 
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Lack  of  human-robot  design  rigor  can  lead  to  catastrophi 
results 


•  Catastrophic  problems  would  result  in  severe  setbacks  to  the 
fielding  of  robotic  systems 

•  No  “user-pull”  for  semi/autonomous  systems  to  couple  user 
needs  with  research 

•  Robotics  communities  are  fragmented,  no  advocate  or 
manager  for  robotics  technology 


ASB  Studv  Recommendation 
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Create  a  new  “systems-oriented  program  for  analysis, 
understanding,  development  and  improvement  of  human-robot 
interactions”  with  ARL  as  program  steward  (with  other  agency 
j;ooperatioh),  stimulating  spiral  development 

Requirements  Community  should: 

•  establish  operational  architecture  for  autonomous  robots 

•  validate  witlh  available  field  testing 

FCS  Increment  I  should  have  as  a  minimum: 

•  follower  robots  w /  significant  level  of  autonomy 

•  surveillance  and  reconnaissance  robots  operating  in  limited 
environment 

S  J' 
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Baseline  SMI 


System  Integration 
(Lab} 


Demo:  *0 
Semi-Autonomous 
DevfHoprneht 


Crew  integrationand  Automation  (CAT) 
Robotic'  F  ollowef  (RF)  ATDs.-' 
Waifigtiter  Experiment  Fort  Jiliss 


IVES  STO 
ield  Experiment 


CAT andRF  AIDS  S 

Final  Warfighter  Experiment 

: 

Hyman  Robot 

i  „  ..  ,  interaction  STO 


TARDEC  Program  Progress 


Robotic  Technology 
Development 


U.S.  ARMY  TANK-AUTOMOTIVE  RESEARCH  DEVELOPMENT  AND  ENGINEERING  CENTER 


■  Y93 

i 

CrewmaJv^^^ 

Associate 

FY96 

_  FY98  tSfl 
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Focus  is  Integration 
VTI  Program 

-  Crew  integration  and  Automation 
Testbed  (CAT)  ATD 


Backqround 
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Crew  integration  &  Automation  Testbed 
(CAT)  ATD 


Technologies 

•  Crew  Driving  and  Decision  Aids 

•  Advanced  Warfighter  Interfaces  (AWI) 

•  UGV,  small  UGV,  and  UAV  Control 

•  Multi-mission  Crew  stations 

•  Autonomous  Navigation  for  MGV 

•  Embedded  Simulation  System  Warfighter  Payoff 

•  Enhance  performance  and  minimize  workload  to 
support  reduced  crew  size 

•  Control  various  unmanned  systems  from  a  common 
crew  station  interface 

•  Mission  planning  and  rehearsal  while  deployed  with 
embedded  simulation 

•  Develop  TTPs  for  unmanned  systems  through 
continual  field  experiments 

Demonstrating  the  crew  interfaces,  automation,  and 
integration  technologies  for  Current  and  Future  Systems 
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Crewstation  Display  Hardware 
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COTS  Sharp  20.1”  TFT-LCD  display  was 
selected  due  to  video  requirements 

-  Resolution:  1600X  1200 

-  Optical  Response:  5  ms  ON,  20  ms  OFF 

Portrait  Orientation 

-  Allows  up  to  2  “SMI  displays”  per  display 


•  3  displays  per  crewstation 

-  Combined  1 35°  HFOV  (45°  each) 


•  Two  side  displays  were  angled  for  equal 
viewing  distance  to  each  panel 

•  Goal:  Seamless  gap  between  displays 
for  indirect  vision  imagery 

-  Display  and  touchscreen  hardware  resulted 
in  2”  gap  between  displays  (1”  around  each 
display) 
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Crew  Station  Features  - 
Screen  Functionality 

A  SUPERIOR  ARMY 


ARV  Drive 
Tactical  Map 


Target  Acq  Sensor 
&  Unmanned  Asset  Control 
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Edit 

Symbol 


Crew  Station  Features 
Multi-modal  Inputs 


Multi-modal  Interface 


INDIRECT 

FIRES 


MAP 


Redundant  Inputs 

-  Hard  (bezel)  buttons 

-  Touch  buttons 

•  Button  type  indicators  can  be 
used  to  anticipate  button 
behavior. 

-  Yoke 

-  Voice  commands  _ 

-  Keyboard/Trackball 


MSN 

PLNG 
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Embedded  Simulation  System 


MISSION  APPLICATIONS 

•  Embedded  Training 

•  Mission  Rehearsal 

•  Mission  Planning 
SIMULATION  CAPABILITIES 

•  Simulated  Turret 

•  Virtual  Lethality 

•  Virtual  Sensors 


Crew  Stations 


Vehicle 


Vehicle 
and  crew 
interaction 
data 


Simulated  ATR 


•  Simulated  ATT 

•  Simulated  C2 

VEHICLE  SIMULATIONS 

•  Mobility 

•  Survivability 

•  Virtual  OPFOR 

•  Virtual  Friendlies 

OPERATIONAL  APPLICATIONS 


Embedded 

Simulation 

System 


Virtual  Battlefield 


Battlefield  Visualization 
Terrain  Registration 
Virtual  Sensor  Coverage 
Virtual  Lethality  Coverage 
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HRI  Program  Methodology 

SUPERIOR  TECHNOLOGY  FOR  A  SUPERIOR  ARMY 

Program  Methodology 

•  Requirements  analysis/Task  decomposition 

•  Ontology/Behaviors  development 

•  Modeling  Environment 

-  End-to-end  modeling  environment 

-  Constructive  modeling  and  simulation 

-  WMI  decomposition 
j-  Component/system/vehicle  modeling 

-  Virtual  and  HWITL  simulation 

mKSm 

•  Technology  Exploration 

-  Multi-model  devices 

-  Scalable  interfaces 
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chnology  for  Human-Robot  Interaction  (HRif 
in  Soldier-Robot  Teaming 
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Goa/: 

Provide  intelligent,  scalable  mounted  and 
dismounted  control  for  unmanned  ground 
and  air  systems  and  optimize  human-robot 
teams 


Pacing  Technologies : 

•  Human-robot  teams 

•  Intelligent  scalable  interface 
•Intelligent  agents  and  adaptive 
automation 


•Recursive  end  to  end  modeling 
environment 
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What  HRI  provides  for  the  Warfighter 
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•  I  Reduces  training/retraining  burden  between  mounted  and  dismounted 

controlling  missions 

•  Reduces  task  timelines 

•  Eases  cognitive  burden  on  soldier 

•  Provides  human-centered  design 

•  Standardizes  air  and  ground  unmanned  systems  interfaces 

•  Provides  scalability  for  varying  screen  sizes 

•  Sheds  tasks  when  soldier  is  overloaded,  adds  tasks  to  keep  soldier  alert 

•  Consolidates  Army  interface  programs 

•  Optimizes  soldier-robot  teaming 


CAT 


AID 


FFW 


HRI 


Soldier  Focused 


•Reduce  Controlling  Workload 
•Optimize  Teaming  w /  vehicle 
•Scale  SMI  for  Mounted  &  Dismounted  Ops 
•Provide  like  control  for  UGV’s  &  UAV’s 


Scalable 

Interface 

Intelligent 

Agents 

Adaptive 

Automation 
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Vehicle  Focused 

•Increase  current  perception  capabilities 
•Make  vehicle  more  survivable 
•Address  anti-tampering  issues 
•Provide  tactical  behaviors 


Tactical 

Behaviors 


HRI  Knowledgebase 


•Tasks,  behaviors 
•Models 

•Component 

•System 

•Vehicle 

•Repository 

•Data 

•Software 


Virtual  Simulation 
•ESS  <- 

•IWARS 
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Simulator 


LOGY  FOR  A  SUPERIOR  ARMY 


►/  Aaent 


Workflow  Execution  Engine 


Resource 
Data  Collection 
and  Reporting 


•Documentation^^® 

Nv  A 

L  A 

>4,  l 

r  \t 

Environment  Interface 


SA  Knowledge  Repository 


Candidate  Workflow  Execution  Engines 

•  Microsaint  (IMPRINT) 

•  Cougar 
•JESS 

•  FCS  TIN  Services 
•VTI  DSS 
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Scalability  Issue  for  Interface 
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T 


1 


0.4  0.5 

Level  of  Control 


□  Wearable  ■  OCU  B  Display  □  Crewstation 
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Intelligent  Scalable  Interface  Builder 


MCS 
mission 
and  tasks 


Interrogate  HW 


Hardware  Systems  \ 

■Crewstation,  Workstatioi 
■Tablet,  PDA,  Wearable 
■HMD’s 

Joysticks,  yokes  / 


HW  Results 

Display  surface 
size,  devices,  etc 


Environment 
•Temperature 
•Time  of  day 
•Terrain 
•Constraints 
(ex  MOPP 
gloves) 


Ml  repository/ 

lelligent  Agent^ 
idgets 

^  •  behaviors 
f  ‘representations 


Configure  WMI 


I—  LJ  L_ 

u  e 


SOLDIER  PREFERENCES 

•Role/security  (MCS  commander,  access 
levels,  comms  config,  etc.) 

•Physical  characteristics  (i.e.  left  handed, 
health,  glasses,  etc) 

•Mental  (thresholds,  baseline) 
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Scalable  Interface  Configuration  hriato-d 

Approach 
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•-  Add  Graptoc  -  Povn 

y-iy»-0y~L 

J»£»[ 

Unlbdy-i 

y-yy-iiy-L 
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•Reduce  training  between  systems 
•Standardize  interface 


•Inputs  are  consistent  (i.e.:  CTRL-C  is  copy) 

•Behavior  is  consistent  ((ex:  button  highlights  when  touched) 
•Intuitive  to  user  -  in  his/her  mission  language 
•Steps  to  do  task  match  TTP’s 
•Present  information  consistently 

•Look  and  feel  (same  font,  color  scheme,  etc) 

•Menu  system  layering 
•Acronyms  are  identical 

•Establish  common  unmanned  system  tasks  (ground  and  air) 

•Mobility,  navigation 


•RSTA 


•Fire  Control 
•communication  link 
•Other 

•Target  processes  to  automate 

•Reduce/eliminate  controlling  aspect  of  mission  to  allow  soldier 
to  focus  on  primary  mission 
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HRI  Program 


SUPERIOR  TECHNOLOGY  FOR  A  SUPERIOR  ARMY 


Systems  Engineering  Approach 

•  Technology  exploration 

-  Multi-model  devices 

-  interfaces 

•  Modeling  environment 

-  Task  decomposition 

-  Behaviors 

-  Constructive,  virtual  HWITL 
simulation 

-  Logical  integration  points 

•  Laboratory  facility 

-  Recursive  simulation 

-  Hardware  trades 
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Methodology 


Aaent  Development 


•  CAT  &  FFW 


-  Architecture  development 

-  Workload  analysis 

-  Intelligent  agent  development 


Layer  Component  Architecture 


Warfighter  ^ 
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Mil 

Model 


Active 

Task 

Set 
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Alternate  Approach 
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SOAR  Tech  Contract 

-  Adjustable  Autonomy 

-  Extensive  modeling 

-  Intelligent  Agent 
development 


Adjustable  Autonomy  Module 


ivrnxeo- 
Initiative 
Inter  actio  r 
Reasoner 


Bayesian 

Task 


Human 

Input 


Recognizer 


Human 

Performance 

Architecture 


Task 

Mocels 


Current 

Situation 


Device 

Mace! 


World  Events 


lysiological  Monitoring  Methods 


Heart  rate 


skin  changes 
brain 


patterns 
*Eye  tracking 

Other  Monitoring  Methods 
*Task  execution  times 
•Error  identification 
•Observation 


Soldier 


A  u  to  rna_te/ Ad ju  s_t 


Decision  aids 


Task  automation 


SIV11  design  enhancements 
Vehicle  tactical  behaviors 


Sensor  advancements 


WMI/System 


Other 


soldiers 


automation  levels 


Other 

battlefield 

entities 
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End  to  End  Collaborative  Modeling  Environment 

Constructive  Virtual  Live  Distributed 


ART 

STO 


Payload 

Modeling  ‘Sensors 
•Comms 


•Survivability 

•Etc. 


Unmanned 
Systems 
Modeling 


Tiffing  & 

Adaptive 

Automation 


VSIL,  WMI, 
Motion  Sim 


Inputs: 

•Requirements 
•AUTL’s,  TTP’s,  FOC’s 
•Task  analysis 
•Framework 
•Architecture 
•Cognitive 
•CAT  &  RF 
•FCS/FFW 

•Other  joint  programs 


Repository 

•Models 

•Documents 

•Data 


TARDEC,  ARL  & 

l 

Manned  &  Unmann 


/ 


Optimize: 

•Task,  platform,  payload  automation  mix 

•Soldier  workload 

•Rapid  WMI  prototyping 

•Architecture,  components,  processing, 
modularity 


Technology  readiness  levels 


Technology  Exploration 
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Optimal  Multi-modal  input  combinations 

•  Mission  (hardware)  dependent 

-  Mounted  (large  display  area) 

-  Dismounted  (limited  display  area) 

•  Potential  technologies 

-  Speech  recognition 

-  Haptic,  vibro-tactile 

-  HMD’s,  ocular,  display  size  scalability 

-  3-D  audio,  head  tracker  (comms,  alerts,  etc.)  v-v-V, 

-  Biometrics  (i.e  card  reader,  user  preferences) 

-  Soldier  monitoring  systems  (workload,  stress) 

-  Joystick,  yoke,  force  feedback 

-  Face  recognition 

-  Eye  tracking 

-  Gesturing 


0  £3 


INDIRECT 


TARGET 


MAP 


FIRES 


ACQ 


ARGET 


MSN 

PLNG 


RSTA 


JEUE 
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